Purpose

Enable Qwen3.5/Qwen3Next models whose attention, GDN, and dense-MLP dimensions are not evenly divisible by tensor parallel size. The tested target is dense Qwen3.5 27B NVFP4 with TP=3 and DCP=3 on three PCIe Blackwell GPUs.

This replaces the previous broad experimental branch with a clean rebase onto current main. MTP, DFlash, MoE/EP experiments, low-bit collectives, diagnostics, profiling hooks, and rejected POCs are intentionally excluded.

Approach

• Add explicit uneven GQA and Gated DeltaNet head partitions with padded local projections and logical checkpoint spans.
• Add padded merged-column and row-parallel weight loaders, including NVFP4/WNA16 packed dimensions and scalar scales.
• Pad the dense Qwen MLP intermediate dimension while preserving grouped W4A16 alignment (17408 -> 17472 for TP=3).
• Size Mamba states and CUDA-graph profiling caches from the actual uneven/DCP geometry.
• Align Mamba prefill chunks to effective DCP block geometry.
• Skip construction and loading of the Qwen3.5 visual tower under --language-model-only.
• Pad and mask the CP LSE correction reduction for non-power-of-two world sizes (N=3, launch width 4).

The generalized DCP validation change is restricted to qwen3_5_text and qwen3_next; other GQA/MQA models retain the previous fail-fast behavior.

Dependencies and duplicate-work check

Searches were run for TP3 Qwen3.5, uneven tensor parallelism, FlashInfer DCP metadata, non-power-of-two DCP, and concurrent partial prefill.

• No open PR was found for the uneven Qwen3.5/Qwen3Next TP3 layout work or the non-power-of-two CP correction.
• [Bugfix] Fix corrupted output with FlashInfer + decode context parallelism #48003 independently fixes FlashInfer rank-local paged-KV metadata more completely. The duplicate implementation was removed from this PR. This branch requires [Bugfix] Fix corrupted output with FlashInfer + decode context parallelism #48003 (or equivalent) for correct FlashInfer+DCP paged metadata.
• [Core] V1 Concurrent Partial Prefills Enablement #41399 and [Core] Concurrent partial prefills for V1 #31330 implement V1 concurrent partial prefill. The duplicate guard removal was removed from this PR; partial4 was validated only with a local runtime override.

Tests

28 focused pytest tests passed
Ruff passed for every changed Python file
git diff --check passed
direct CUDA DCP3 LSE probe passed for natural-log and base-2 LSE

The focused suite covers uneven attention/GDN partitions, packed NVFP4/WNA16 loaders, padded vocab/MLP shards, Mamba DCP chunk alignment, and CUDA-graph KV sizing.

Model and serving validation

Candidate: nvidia/Qwen3.6-27B-NVFP4, TP=3, DCP=3, FlashInfer 0.6.14, FP8 KV, prefix caching, eager disabled, FULL+PIECEWISE CUDA graphs, no MTP.

• Server initialized successfully and captured all configured graphs.
• GPU KV cache: 412,903 logical tokens.
• Maximum concurrency at 153,600 tokens/request: 2.69x with memory utilization 0.82.
• Graph memory: approximately 0.51 GiB per GPU.
• Tool-call sanity returned finish_reason=tool_calls, the requested function, and valid JSON arguments in repeated probes.
• Partial4 local POC: 4/4 unique long prompts succeeded, 0 errors, 1122.65 prompt tok/s, wall/p50 about 22.7s.
• Single-stream decode observed on the ABI-overlay candidate: 12.8 tok/s. This is a correctness sanity result, not a final performance claim, because the older binary selected Marlin weight-only FP4.

MTP is excluded because the controlled agent workload regressed from 8/8 in 228.3s without MTP to 7/8 in 294.6s with MTP, despite synthetic decode gains.

AI assistance

AI assistance was used for research, implementation, testing, and preparation of this PR. The submitter reviewed the resulting scope and is responsible for the change end to end.